Emotion-Based Music Visualization Using Photos

Music players for personal computers are often featured with music visualization by generating animated patterns according to the music’s low-level features such as loudness and spectrum. This paper proposes an emotion-based music player which synchronizes visualization (photos) with music based on the emotions evoked by auditory stimulus of music and visual content of visualization. For emotion detection from photos, we collected 398 photos with their emotions annotated by 496 users through the web. With these annotations, a Bayesian classification method is proposed for automatic photo emotion detection. For emotion detection from music, we adopt an existing method. Finally, for composition of music and photos, in addition to matching high-level emotions, we also consider low-level feature harmony and temporal visual coherence. It is formulated as an optimization problem and solved by a greedy algorithm. Subjective evaluation shows emotion-based music visualization enriches users’ listening experiences.